feat: wire threat-model.md into Phase 4 validation planning by pruiz · Pull Request #52 · pruiz/CodeCome

pruiz · 2026-06-09T23:19:41Z

Closes #35.

Summary

Adds explicit itemdb/notes/threat-model.md awareness to Phase 4 validation planning, mirroring the integration already present in Phase 2 (auditor) and Phase 3 (reviewer).

Changes

File	What changed
`prompts/phase-4-validate.md`	Added threat-model.md to required reading; added step 5 (threat-model-aware validation planning) with renumbered workflow; added final-summary item for threat-model assumptions
`.opencode/agents/validator.md`	Added threat-model.md to required reading; added step 5 (threat-model alignment check) with renumbered workflow
`.opencode/skills/exploit-validation/SKILL.md`	Added threat-model.md to inputs; added step 4 (threat-model alignment review) with renumbered workflow
`templates/evidence-readme.md`	Added `# Threat-model assumptions (if applicable)` section between Environment and Commands
`tests/test_phase_1_prompts_threat_model.py`	Added 5 Phase 4 prompt tests (reference, conditional language, attacker capabilities, trust boundaries, existing controls)

Acceptance criteria

Phase 4 prompt explicitly references itemdb/notes/threat-model.md when present
Validation plans avoid assuming attacker capabilities contradicted by the threat model
Validation evidence or notes mention material threat-model assumptions when they affect the result
Existing Phase 4 behavior remains compatible with projects that do not yet have threat-model artifacts (all references use "when available" / "when present" language)
All 780 tests pass
Frontmatter and artifact checks clean

Summary by CodeRabbit

Documentation
- Validation workflow now inserts a threat-model alignment step before source inspection.
- Guidance and checklists updated to instruct validators to consult the threat model when available.
- Evidence template includes a “Threat-model assumptions (if applicable)” section.
Tests
- Added tests to verify threat-model references and required phrasing (conditional availability, attacker capabilities/non-capabilities, trust boundaries, existing controls).

Add explicit threat-model.md awareness to the Phase 4 validator agent, exploit-validation skill, prompt, and evidence template. Mirror the existing Phase 2/3 integration pattern: - Required reading references threat-model.md (conditional: when available). - Validation workflow includes a threat-model consultation step covering attacker capabilities/non-capabilities, trust boundaries, existing controls, affected assets, and open assumptions. - Evidence README template gains a dedicated Threat-model assumptions section. - Final response summary includes threat-model assumptions that materially affected validation. 5 new prompt tests assert the Phase 4 prompt references threat-model.md, attacker capabilities/non-capabilities, trust boundaries, existing controls, and uses conditional language. Closes #35.

coderabbitai · 2026-06-09T23:19:51Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 6ee35132-b8a4-477b-b2f5-398d076912ad

📥 Commits

Reviewing files that changed from the base of the PR and between 3aca7d2 and 9fae198.

📒 Files selected for processing (1)

tests/test_prompts_threat_model.py

🚧 Files skipped from review as they are similar to previous changes (1)

tests/test_prompts_threat_model.py

📝 Walkthrough

Walkthrough

Phase 4 validation planning now consults itemdb/notes/threat-model.md when available, adding a threat-model alignment step before source inspection. Evidence templates capture any material threat-model assumptions, validator and skill docs reference the threat model, and tests verify prompt and documentation updates.

Changes

Threat-Model Integration in Phase 4 Validation

Layer / File(s)	Summary
Core threat-model prompt integration `prompts/phase-4-validate.md`	The phase-4-validate.md prompt adds a conditional instruction to consult `itemdb/notes/threat-model.md` when available, inserts a new threat-model alignment review step in the validation workflow before source-code inspection (reviewing attacker profile, trust boundaries, existing controls, and affected assets), and extends the final response checklist to require documenting threat-model assumptions that materially affected validation strategy or evidence interpretation.
Evidence template threat-model section `templates/evidence-readme.md`	A new optional "Threat-model assumptions (if applicable)" section provides guidance for documenting threat-model assumptions that affected validation outcomes, with example bullets and a note that the section may be omitted if the threat model did not influence the result.
Validator and skill documentation alignment `.opencode/agents/validator.md`, `.opencode/skills/exploit-validation/SKILL.md`	The validator.md agent guide and exploit-validation SKILL.md are updated to reference `itemdb/notes/threat-model.md` as an optional required reading/input when present, and both add a new threat-model alignment review step to their validation workflow sections (checking that attacker profile, trust boundary, and impact align with the threat model).
Threat-model integration test suite `tests/test_prompts_threat_model.py`	New test utilities and Phase 4 assertions verify that `phase-4-validate.md`, validator.md, and SKILL.md explicitly reference `itemdb/notes/threat-model.md` when present, use conditional availability wording, mention attacker capabilities and explicit non-capabilities, and that `phase-4-validate.md` additionally references trust boundaries and existing controls.

🎯 2 (Simple) | ⏱️ ~12 minutes

A validator's compass now points true,
With threat-models guiding each test through,
From capabilities clear to controls in place,
We validate with rigor and grace. 🐰✨

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the main change: wiring the threat-model.md into Phase 4 validation planning, which is exactly what the pull request implements.
Linked Issues check	✅ Passed	All requirements from issue `#35` are met: Phase 4 prompts reference threat-model.md conditionally, validation planning accounts for attacker capabilities/non-capabilities, trust boundaries, existing controls, assumptions are documented, and backward compatibility is maintained.
Out of Scope Changes check	✅ Passed	All changes are directly scoped to issue `#35` objectives: integrating threat-model.md into Phase 4 validation planning across prompts, documentation, and tests with no unrelated modifications.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

Create stacked PR
Commit on current branch

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch feat/issue-35-phase-4-threat-model

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

github-actions · 2026-06-09T23:21:28Z

Coverage Report

Metric	Value
Line Coverage	75.7%
Lines Covered	0 / 0

Download detailed HTML coverage reports per OS/Python from the workflow artifacts.

Generated by pytest-cov on 2026-06-09T23:53:00.290Z

Copilot

Pull request overview

This PR wires itemdb/notes/threat-model.md into Phase 4 validation guidance (prompt/agent/skill) and updates the evidence template + tests so validation planning and reporting can incorporate threat-model constraints and assumptions, consistent with prior Phase 2/3 integrations.

Changes:

Extend Phase 4 validation workflow to incorporate threat-model alignment checks (capabilities/non-capabilities, trust boundaries, existing controls, assumptions).
Add an evidence template section for documenting threat-model assumptions when they materially affect validation.
Add Phase 4 prompt-level tests to enforce threat-model references and key terminology.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
`prompts/phase-4-validate.md`	Adds threat-model-aware validation planning step and final summary note.
`.opencode/agents/validator.md`	Updates validator agent required reading + workflow to include threat-model alignment.
`.opencode/skills/exploit-validation/SKILL.md`	Updates exploit-validation skill inputs + workflow to include threat-model alignment review.
`templates/evidence-readme.md`	Adds a “Threat-model assumptions (if applicable)” section to evidence README structure.
`tests/test_phase_1_prompts_threat_model.py`	Adds tests asserting Phase 4 prompt references and threat-model-related wording.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

greptile-apps · 2026-06-09T23:22:54Z

Greptile Summary

This PR wires itemdb/notes/threat-model.md into Phase 4 validation by adding conditional reads, a new workflow step, and a summary bullet to three Phase 4 entry-points, mirroring the pattern already established in Phases 2 and 3. It also renames the test file from test_phase_1_prompts_threat_model.py to test_prompts_threat_model.py and adds coverage for all three modified files.

prompts/phase-4-validate.md gains a detailed step 5 with five sub-bullets (attacker capabilities, trust boundaries, existing controls, affected assets, open assumptions) and a final-summary item for threat-model assumptions.
.opencode/agents/validator.md and .opencode/skills/exploit-validation/SKILL.md each get a lighter-weight threat-model alignment step renumbering the workflow, keeping the pattern consistent across entry-points.
tests/test_prompts_threat_model.py (renamed) adds nine new assertions covering threat-model references and conditional language in all three modified Phase 4 files.

Confidence Score: 5/5

All changes are additive documentation and test additions with no behavioural or configuration logic; safe to merge.

Every modified file is a prompt, template, or test — no executable code paths are changed. The new workflow steps are conditional ('if present' / 'when available'), preserving backward compatibility for projects without a threat model. Tests added for all three modified Phase 4 files confirm the key strings are present. No regressions are introduced.

No files require special attention.

Important Files Changed

Filename	Overview
prompts/phase-4-validate.md	Added threat-model.md to required reading and a detailed step 5 (5 sub-bullets) covering attacker capabilities, trust boundaries, existing controls, assets, and open assumptions; workflow renumbered and final-summary updated consistently.
.opencode/agents/validator.md	Added conditional threat-model read to the required-reading list and a new step 5 for threat-model alignment; workflow renumbered from 11 to 12 steps with no conflicts.
.opencode/skills/exploit-validation/SKILL.md	Added threat-model.md to inputs and a new step 4 for attacker-profile/trust-boundary alignment; step 4 omits 'open assumptions' sub-point present in the main prompt, but this appears intentional given the skill's tighter scope.
templates/evidence-readme.md	Added an optional 'Threat-model assumptions' section with four example bullets; clearly marked as omittable when the threat model did not affect the result.
tests/test_prompts_threat_model.py	Renamed from test_phase_1_prompts_threat_model.py; adds nine new Phase 4 assertions covering validator.md, SKILL.md, and phase-4-validate.md — threat-model reference, conditional language, attacker capabilities, trust boundaries, and existing controls.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A([Start Phase 4 Validation]) --> B[Read finding, source files,\nsandbox docs]
    B --> C{itemdb/notes/threat-model.md\npresent?}
    C -- Yes --> D[Step 5: Review threat-model alignment\n- Attacker capabilities / non-capabilities\n- Trust boundaries\n- Existing controls\n- Affected assets\n- Open assumptions]
    C -- No --> E[Skip threat-model step]
    D --> F[Inspect source files]
    E --> F
    F --> G[Prepare sandbox]
    G --> H[Execute / improve validation plan]
    H --> I[Capture commands, inputs, outputs]
    I --> J[Store evidence under itemdb/evidence/]
    J --> K[Update finding]
    K --> L[Move finding to correct status dir]
    L --> M[Final summary\n+ threat-model assumptions\nthat affected strategy]

_{Reviews (3): Last reviewed commit: "fix: strengthen test_exploit_validation_..." | Re-trigger Greptile}

- Make threat-model bullets explicitly optional ("If present, …") in prompt, validator agent, and exploit-validation skill per Copilot review. - Rename test_phase_1_prompts_threat_model.py -> test_prompts_threat_model.py to reflect multi-phase coverage per Greptile review. - Add 4 new tests covering validator.md and SKILL.md threat-model additions. - Update conditional-language test to also accept "if present" phrasing.

coderabbitai

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/test_prompts_threat_model.py`:
- Around line 136-139: The test
test_exploit_validation_skill_mentions_attacker_capabilities currently only
asserts "non-capabilities" and can miss removal of the phrase "attacker
capabilities"; update the test to check for both keywords (case-insensitive) by
reading the SKILL.md content and asserting that it contains "non-capabilities"
and also contains "attacker capabilities" (or the exact phrase your
documentation uses) so both are validated; locate the assertion in
test_exploit_validation_skill_mentions_attacker_capabilities and add the second
assertion (or a combined check) against the variable content read from
".opencode/skills/exploit-validation/SKILL.md".

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 3946fd19-ae22-426a-9f1e-361d3cc7a0e0

📥 Commits

Reviewing files that changed from the base of the PR and between 49f3ab1 and 3aca7d2.

📒 Files selected for processing (4)

.opencode/agents/validator.md
.opencode/skills/exploit-validation/SKILL.md
prompts/phase-4-validate.md
tests/test_prompts_threat_model.py

✅ Files skipped from review due to trivial changes (2)

.opencode/skills/exploit-validation/SKILL.md
.opencode/agents/validator.md

🚧 Files skipped from review as they are similar to previous changes (1)

prompts/phase-4-validate.md

pruiz · 2026-06-09T23:39:33Z

All review feedback addressed in 3aca7d2:

Copilot — make threat-model bullets explicitly optional (3 threads):

prompts/phase-4-validate.md:40: Changed to If present, ... — …
.opencode/agents/validator.md:27: Changed to If present, ... — …
.opencode/skills/exploit-validation/SKILL.md:33: Changed to If present, ... — …

Greptile — test file naming:

Renamed test_phase_1_prompts_threat_model.py → test_prompts_threat_model.py

Greptile — missing tests for validator.md / SKILL.md:

Added _read_opencode helper + 4 new tests covering validator.md and SKILL.md

CodeRabbit — docstring coverage: Skipped — project convention (no existing test functions in this file have docstrings).

784 tests pass, frontmatter and artifact checks clean.

pruiz

Resolved all review feedback in 3aca7d2:

Copilot (3 threads — bullets not explicitly optional)

prompts/phase-4-validate.md:40: Changed to If present, itemdb/notes/threat-model.md — … so a missing file won't cause agents to fail.

.opencode/agents/validator.md:27: Same fix — leads with If present, ….

.opencode/skills/exploit-validation/SKILL.md:33: Same fix — leads with If present, ….

Greptile (2 threads — test naming + coverage)

Test file naming: Renamed test_phase_1_prompts_threat_model.py → test_prompts_threat_model.py (the file already covered Phases 2/3/4, not just Phase 1).

Missing tests for validator.md / SKILL.md: Added _read_opencode helper and 4 new tests:

test_validator_agent_references_threat_model — asserts itemdb/notes/threat-model.md in validator.md
test_validator_agent_uses_conditional_language — asserts "if present" / "when available" phrasing
test_exploit_validation_skill_references_threat_model — asserts itemdb/notes/threat-model.md in SKILL.md
test_exploit_validation_skill_mentions_attacker_capabilities — asserts "non-capabilities" in SKILL.md

CodeRabbit (docstring coverage warning)

Skipped — matches project convention (no existing test functions in this file use docstrings).

784 tests pass, frontmatter and artifact checks clean.

…ilities Add assertion for "attacker" keyword alongside existing "non-capabilities" check per CodeRabbit review feedback. The skill step 4 text contains both "attacker profile" and "documented capabilities, non-capabilities".

pruiz requested a review from Copilot June 9, 2026 23:19

Copilot started reviewing on behalf of pruiz June 9, 2026 23:19 View session

Copilot AI reviewed Jun 9, 2026

View reviewed changes

Comment thread prompts/phase-4-validate.md Outdated

Comment thread .opencode/agents/validator.md Outdated

Comment thread .opencode/skills/exploit-validation/SKILL.md Outdated

greptile-apps Bot reviewed Jun 9, 2026

View reviewed changes

Comment thread tests/test_prompts_threat_model.py

Comment thread tests/test_prompts_threat_model.py

coderabbitai Bot reviewed Jun 9, 2026

View reviewed changes

Comment thread tests/test_prompts_threat_model.py